Don't build C source if running on PYPY#5
Conversation
Avoid building C optimised version if running on PYPY and use the plain python implementation 1- pypy have a penalty cost at the python to C interface 2- I found this memory leak: pypy/pypy#5100 As implementation I took inspiration from msgpack https://github.com/msgpack/msgpack-python/blob/main/setup.py
|
Unfortunately the performance hit from using pure-Python version is massive: import timeit
import os
if "PURE_PYTHON" in os.environ:
from cobs.cobs import _cobs_py as cobs
else:
from cobs import cobs
with open("/vmlinuz", "rb") as f:
ddata = f.read()
edata = cobs.encode(ddata)
number = int(os.environ.get("ITERS", "100"))
dtime = timeit.timeit(lambda: cobs.encode(ddata), number=number) / number / len(ddata)
etime = timeit.timeit(lambda: cobs.decode(edata), number=number) / number / len(edata)
print(f"encoding: {1 / (dtime * 1e9)} GB/s")
print(f"decoding: {1 / (etime * 1e9)} GB/s")For many applications, such as mine, this would make the package effectively useless. |
|
I wonder if it would be possible to set up the package so that the C extension is an installation option. I'm only vaguely aware of a possible mechanism: in the qrcode package, it mentions the ability to install it with an optional dependency:
|
As far as I'm aware, this mechanism can't be used in the way you suggest. Ultimately, I think it's OK to leave the C extension to be built when installing on PyPy: it does improve performance (if I recall correctly by a factor of about 2x) despite the fact that on PyPy, the Python C API bridge is relatively slow and the use of CFFI is preferred. |
|
I would prefer to solve this the "proper way", but I don't know what the proper way is. I've looked at this several times over the years. It is hard to come up with a solution that satisfies all use-cases automatically. Every use-case can be worked out manually, but automatic is preferred. Some projects in the past have written the I could split the cobs package into two separate packages, something like |
Avoid building C optimised version if running on PYPY and use the plain python implementation
As implementation I took inspiration from msgpack https://github.com/msgpack/msgpack-python/blob/main/setup.py
Alternative is to use
import cobs.cobs._cobs_py as cobsinstead offrom cobs import cobsbut this move the burden to the userspace and it is pretty uglyThanks,
Luca